Noisy Expectation-Maximization: Applications and Generalizations

We present a noise-injected version of the Expectation-Maximization (EM) algorithm: the Noisy Expectation Maximization (NEM) algorithm. The NEM algorithm uses noise to speed up the convergence of the EM algorithm. The NEM theorem shows that injected noise speeds up the average convergence of the EM algorithm to a local maximum of the likelihood surface if a positivity condition holds. The generalized form of the noisy expectation-maximization (NEM) algorithm allow for arbitrary modes of noise injection including adding and multiplying noise to the data. We demonstrate these noise benefits on EM algorithms for the Gaussian mixture model (GMM) with both additive and multiplicative NEM noise injection. A separate theorem (not presented here) shows that the noise benefit for independent identically distributed additive noise decreases with sample size in mixture models. This theorem implies that the noise benefit is most pronounced if the data is sparse. Injecting blind noise only slowed convergence. I. NOISE BOOSTING THE EXPECTATION-MAXIMIZATION ALGORITHM We show how carefully chosen and injected noise can speed convergence of the popular expectation-maximization (EM) algorithm. A general theorem allows arbitrary modes of combining signal and noise to improve the speed of parameter estimation. The result still speeds EM convergence on average at each iteration so long as the injected noise satisfies a positivity condition. The EM algorithm generalizes maximum-likelihood estimation to the case of missing or corrupted data [1], [2]. Maximum likelihood maximizes the conditional signal probability density function (pdf) f(y|θ) for a random signal variable Y given a vector of parameters θ. It equally maximizes the log-likelihood ln f(y|θ) since the logarithm is monotone increasing. So the maximum–likelihood estimate θ∗ is